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Executive  Summary 

Image  segmentation  is  a  process  to  extract  and  organize  information  energy  in  the 
image  pixel  space  according  to  a  prescribed  feature  set.  It  is  often  a  key  preprocess 
in  automatic  target  recognition  (ATR)  algorithms.  In  many  cases,  the  performance  of 
image  segmentation  algorithms  will  have  significant  impact  on  the  performance  of  ATR 
algorithms.  Due  to  the  variations  in  feature  set  dehnitions  and  the  innovations  in  the 
segmentation  processes,  there  is  large  number  of  image  segmentation  algorithms  existing 
in  the  ATR  world.  The  problem  is  which  image  segmentation  algorithm  performs  best 
for  an  ATR  application.  There  are  a  number  of  measures  to  evaluate  the  performance 
of  segmentation  algorithms,  such  as  Percentage  Pixels  Same  (pps),  Partial  Directed 
Hausdorff  (pdh)  and  Complex  Inner  Product  (cip).  In  the  research,  we  found  that  the 
combination  of  the  three  measures  shows  effectiveness  in  the  evaluation  of  segmentation 
algorithms  against  truth  data  (human  master  segmentation).  However,  we  don’t  know 
what  are  the  impact  of  those  measures  in  the  performance  of  ATR  algorithms  that 
are  commonly  measured  by  Probability  of  detection  (Poet);  Probability  of  false  alarm 
{Pfa)i  Probability  of  identification  (Pid),  etc.  In  all  practical  situations,  ATR  boxes 
are  implemented  without  human  observer  in  the  loop.  The  performance  of  synthetic 
aperture  radar  (SAR)  image  segmentation  should  be  evaluated  in  the  context  of  ATR 
rather  than  human  observers. 

In  this  research,  a  limited  literature  search  has  been  conducted.  As  we  expected,  there 
have  been  no  apparent  research  efforts  associated  to  the  evaluation  of  image  segmentation 
algorithms  for  ATR  in  the  literature.  Most  of  the  image  segmentation  evaluation  meth¬ 
ods  are  limited  to  the  comparison  with  human  observers  or  truth  data.  This  research 
project  establishes  a  preliminary  segmentation  algorithm  evaluation  suite  involving  seg¬ 
mentation  algorithm  performance  measures  as  well  as  the  ATR  algorithm  performance 
measures.  The  suite  includes  a  set  of  baseline  ATR  algorithms  and  standard  ATR  per¬ 
formance  evaluation  measures.  It  provides  a  quantitative  evaluation  method  to  judge 
which  SAR  image  segmentation  algorithm  is  the  best  for  a  particular  ATR  application. 
Preliminary  experiment  results  based  on  some  baseline  ATR  algorithms  and  a  typical 
image  segmentation  algorithm  using  the  evaluation  suite  are  tabulated  in  this  report.  It 
is  our  conclusion  that  using  traditional  evaluation  measures  does  not  necessarily  reflect 
the  performance  and  impact  of  image  segmentation  algorithm  towards  the  ATR  perfor¬ 
mance.  Higher  in  traditional  image  segmentation  performance  score  does  not  guarantee 
the  higher  in  ATR  performance  score. 

A  significant  portion  of  the  research  effort  spend  on  identifying  and  implementing 
a  suite  of  baseline  ATR  algorithms.  To  our  surprise  that  there  is  no  existing  suite  of 
baseline  ATR  algorithms  in  the  ATR  community,  even  though  every  one  is  referring  to 
it.  In  order  to  establish  a  baseline  ATR  algorithms  for  testing,  we  studied  an  existing 
software  on  template  matching  ATR  algorithm  developed  at  AFRL[5]  and  added  two 
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more  algorithms  to  form  a  baseline  ATR  algorithm  suite.  The  additional  algorithms 
are  conditional  Gaussian  model  based  ATR[6]  and  a  SAR  ATR  system  developed  in  the 
Lincoln  Laboratory  [7]. 

The  another  outcome  from  this  research  effort  is  a  road  map  leading  to  further  re¬ 
search  in  this  area.  Even  though,  invoking  ATR  algorithm  suite  is  effective  in  evaluating 
the  image  segmentation  algorithms  in  the  context  of  ATR.  There  are  several  problems: 
a)  The  image  feature  space  of  an  image  segmentation  algorithms  does  not  necessarily 
match  the  region  of  interest  of  the  ATR  algorithms  in  the  suite;  b)  The  uncertainty 
(multiple  thresholds  and  choices  of  parameters)  in  the  ATR  algorithm  implementation; 
c)  It  is  costly  every  time  to  invoking  multiple  ATR  algorithms  in  the  evaluation  of  an 
image  segmentation  algorithm.  A  road  map  for  the  further  research  to  overcome  those 
problems  is  presented  in  this  report. 


1  Introduction 

Image  segmentation  is  often  a  key  step  in  an  automatic  target  recognition  system.  There¬ 
fore,  the  performance  of  image  segmentation  algorithm  is  closely  related  to  the  perfor¬ 
mance  of  an  ATR  system.  In  the  ATR  world,  the  image  segmentation  is  also  referred 
to  as  a  pixel-based  ATR  algorithm  that  labels  the  pixels  in  an  image  as  target  pixels 
(also  known  as  “Point  of  Interest”  (POI))  or  non-target  pixels.  The  object-based  ATR 
algorithm[l]  creates  label  to  an  object-sized  group  of  POIs  that  is  referred  to  as  “Re¬ 
gion  of  Interest”  (ROI).  It  is  clear  that  the  performance  of  object-based  ATR  should 
have  great  dependency  on  the  performance  of  image  segmentation  algorithm.  Evalua¬ 
tion  of  the  performance  of  SAR  image  segmentation  also  encounters  the  difficulty  that 
the  absolute  truth  segmentation  is  often  not  available.  Even  the  manual  segmentations 
(pseudo-truth)  of  the  same  image  by  different  human  experts  are  often  dissimilar.  That 
is  due  to  the  SAR  imagery  generated  by  a  radar  instrument  does  not  match  the  ordi¬ 
nary  human  visual  perception.  SAR  image  segmentation  requires  many  trained  expert 
skills  and  depending  on  some  subjective  judgements.  In  the  ATR  applications,  the  SAR 
image  segmentation  is  done  automatically  as  a  part  of  the  ATR  system  without  human 
in  the  loop.  We  think  the  evaluation  should  be  done  in  the  same  automatic  fashion.  In 
addition,  the  segmentation  performance  should  be  measured  in  terms  of  its  application 
purpose  and  functionality  in  an  ATR  system. 

This  research  effort  is  trying  to  establish  a  research  road  map  leading  to  a  stan¬ 
dard  image  segmentation  algorithm  evaluation  suite  in  the  context  of  ATR.  We  have 
conducted  literature  search  to  see  if  there  are  some  prior  work  in  this  area.  We  have 
implemented  three  baseline  ATR  algorithms  which  use  segmented  ROI  for  object  label. 
Some  preliminary  experiments  were  conducted  to  show  that  using  image  segmentation 
quality  metric  alone  does  not  necessarily  consistent  with  the  quality  of  ATR  performance. 
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2  Image  Segmentation  Quality  Metric 

In  the  literature  search,  we  did  not  come  cross  any  prior  works  on  image  segmentation 
metric  in  the  context  of  ATR  problems.  There  are  a  number  of  image  segmentation 
quality  metric  to  provide  an  objective  measure  of  the  performance  of  image  segmentation 
algorithms  by  comparing  the  segmentation  results  with  truth  data  or  master  segmented 
results.  In  addition,  those  metric  were  created  based  on  the  segmentation  of  visual 
band  optical  images  and  not  for  the  feature  spaces  of  SAR  images.  Some  of  the  image 
segmentation  quality  measures  have  been  introduced  in  the  past  works  [2]  [3]  [4].  For 
the  convenience  of  discussion,  we  make  a  concise  summary  of  those  measures  in  the 
following. 

Percent  Pixels  Scime  (PPS)  is  to  compare  two  images  where  all  the  POIs  are 
classihed  as  “1”  and  background  pixels  are  classihed  as  “0” .  PPS  measures  the  percent¬ 
age  of  pixels  that  are  classihed  the  same  way  between  two  binary  images,  Si  and  S2 
where  Si  is  considered  to  be  truth. 


Y^Si(i,j)S2(i,j) 

When  pps  =  1,  it  indicates  image  Si  =  S2  di  perfect  match.  As  pps  =  0,  it  means  a 
complete  miss  match.  PPS  measure  over  punishes  the  misalignment  of  an  otherwise 
perfect  segmentation. 

Partial  Directed  Hausdorff  (PDH)  measures  the  correctness  of  the  closed  bound¬ 
aries  of  segmentations.  Let  A  =  {uq,  ai,  •  •  • ,  ajv-i}  forms  a  closed  contour  of  a  segmenta¬ 
tion  in  image  Aiand  B  =  {60,  ^>i,  •  •  • ,  ^v-i}  is  a  set  of  contour  vertices  of  a  segmentation 
in  image  S2-  The  partial  directed  Hausdorff  distance  is  dehned  as 

hiAA, B)  =  -  ii||}  (2) 


where  is  the  Ath  ranked  distance.  When  A  =  A  —  1,  the  partial  Hausdorff  distance 
takes  the  maximum  distance  in  the  set  A  that  is  referred  to  as  direct  Hausdorff  distance. 

pdh{6)  =  +  1  ^3^ 

where  <5  is  a  prescribed  threshold  and  A  is  the  result  of  solving  hxiA,  B)  =  6  indicating 
the  total  number(A  +  1)  of  vertices  within  the  distance  threshold  6.  When  pdh{8)  =  1,  it 
means  the  segmentations  are  matched  perfectly.  \ipdh(8)  =  0,it  is  a  complete  mismatch. 
Again  PDH  measure  over  punishes  misalignment.  In  addition,  it  often  requires  re¬ 
sample  the  contours  to  assure  that  both  contours  in  the  comparison  have  the  exactly 
same  number  of  contour  vertices. 
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Complex  Inner  Product  (CIP)  also  measures  the  correctness  of  the  closed  bound¬ 
aries  of  segmentations.  In  this  case,  the  Fourier  descriptor  [2]  is  used  to  represent  the 
contours  for  achieving  some  level  of  invariance  in  contour  scaling,  rotation  and  shifting. 
Let  A(k)  and  B(k)  are  the  Fourier  descriptor  of  contours  in  images  5'i  and  S2  respec¬ 
tively  k  =  0, 1,  •  •  • ,  —  1.  To  avoid  instability,  an  amplitude  modulated  phase-only  filter 

is  used  to  represent  the  Fourier  descriptor  of  the  truth  contour  A{k), 

exp{-ji{A{k)}) 

\A{k)\  +  e 

where  £  is  a  very  small  positive  constant  to  avoid  division  by  zero.  Since  the  phase  of 
Fourier  descriptor,  l{A(k)},  preserves  the  shape  of  the  contour,  the  comparison  of  two 
contours  can  be  represented  by  the  correlation  function. 


Ra,bM 


N 


N-l 


E  B{k)  ■ 

k=0 


exp{-jl{A(k)}) 
\A{k)  \  +  e 


(5) 


If  two  contours  match  perfectly,  RA,B{n)  should  show  a  distinctive  peak.  The  CIP 
measuring  the  peakness  of  RA,B{n)  is  defined  as 


cip  =  1  — 


/  (Af-D-maxntfiA.gjn)}  \ 
1  — liT-.  /  \I  _  tt-.  /  / 


JlO 


J.  1 


(6) 


where  cip  ranges  from  zero  to  one  as  one  for  the  perfect  match  [3]. 

All  the  above  mentioned  measures  can  be  used  together  to  achieve  a  composite  mea¬ 
sure.  However,  there  are  no  clear  ways  to  determine  what  kind  of  composite  measure 
should  be  used  under  what  situation[3].  In  intra-algorithmic  comparison,  the  measures 
are  used  to  compare  the  segmentation  result  with  the  truth  data  that  is  often  a  master 
segmentation  created  by  a  human  analyst.  If  truth  data  is  not  available,  the  segmenta¬ 
tion  result  can  be  compared  with  the  result  generated  by  another  segmentation  algorithm 
using  so-called  inter-algorithmic  comparison  [2].  The  comparison  using  these  measures 
are  limited  to  the  domain  of  point  of  interest  and  have  no  link  to  the  performance  of 
ATR  algorithms. 


3  ATR  Algorithm  Performance  Measures 

Object-based  ATR  algorithms  produce  label  to  a  ROI  that  consists  segmented  group 
of  POIs.  ROIs  have  different  categories.  For  example,  ROI  could  be  a  group  of  POIs 
that  are  extracted  edge  components  or  may  be  a  group  of  bright  pixels  that  are  referred 
to  as  target  scattering  centers.  In  most  of  cases,  ROI  is  just  a  square  block  of  SAR 


4 


image  pixels  that  contain  target  information.  Based  on  the  characteristics  of  ROI,  an 
ATR  algorithm  creates  label  or  labels  and  attached  them  to  the  ROI.  The  performance 
of  the  ATR  algorithms  is  measured  by  a  confusion  matrix[l].  The  number  of  rows 
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Figure  1:  A  typical  confusion  matrix 

depends  on  the  number  of  test  objects.  In  the  ATR  algorithm  evaluation,  a  bin  of  test 
images  of  various  types  of  objects  is  prepared.  Each  time  a  test  image  of  certain  type  of 
object  passing  through  the  ATR  algorithm  under  evaluation,  a  matrix  cell  in  that  row 
corresponding  to  the  type  of  object  will  get  an  increment.  As  to  which  cell  (column) 
getting  increment,  it  depends  upon  the  label  decision  made  by  the  ATR  algorithm.  The 
matrix  cell  Mij  indicates  the  number  of  times,  a  type  i  object  is  labeled  as  type  j.  For 
each  experimental  bin,  a  confusion  matrix  will  be  generated.  In  order  to  compare  the 
performance  of  different  ATR  algorithms  using  the  same  experimental  bin  or  the  same 
ATR  algorithm  using  different  experimental  bins  under  various  operating  conditions,  a 
number  summary  performance  measures  will  be  used  in  the  evaluation.  Here  is  a  short 
list  for  a  number  of  commonly  used  summary  performance  measures  [1]. 

Target  Declaration  Probability  (Ptgt-dec)'  This  number  indicates  to  what  per¬ 
centage  of  the  test  object  images,  the  ATR  algorithm  can  make  a  decision.  The  higher 
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the  percentage,  the  better  the  performance. 


Ptgt—dec 


E  E  M,,- 

i=l,2,3,0,4,5  \j=l,2,3,0,R-ID,NT  , 


(7) 


E 


E 


M 


i=l,2,3,0,4,5  \j=l,2,3,0,R-ID,NT,R-Det 


Detection  Probability  {Poet)’  Poet  is  conditioned  only  on  targets  that  separates 
the  detection  performance  evaluation  on  target  objects  and  non-target  objects.  Higher 
Poet  rneans  the  algorithm  has  higher  ability  to  positively  detect  the  targets. 


Poet  - 


E  E  Mij 

i=l,2,3,0  \j=l,2,3,0,R-ID  J 

E  f  E  M,, 

i=l,2,3,0  \j=l,2,3,0,R-ID,NT  . 


(8) 


A  similar  summary  performance  measure  to  Poet  is  Pg,  Probability  of  Success  that 
includes  the  ability  to  positively  detect  non-target  objects  as  well. 


Ps  = 


E  E  Mij  1  +  Mnt,nt 

i=l,2,3,0  \j=l,2,3,0,R-ID  I 


(9) 


E 


E 


M, 


i=l,2,3,0,4,5  \j=l,2,3,0,R-ID,NT 


O 


Probability  of  False  Alarm  (Pfa)’  It  measures  the  percentage  of  non-target 
objects  falsely  labeled  as  target  objects. 


PpA  - 


E  E 

i=4,5  \j  =  l,2,3,0,R-ID 


Mij 


(10) 


E  E 

i=4,5  \j  =  l,2,3,0,R-ID,NT 


Mij 


A  good  algorithm  should  have  lower  Pfa- 

Identification  Declaration  Probability  (PiD-dec)’  This  measure  summarize  the 
ability  for  an  ATR  algorithm  to  make  decisions  on  object  types. 


PiD-dec  - 


E  E  Mij 

i=l,2,3,0  \j  =  l,2,3,0  J 

E  (  E  Mij 

i=l,2,3,0  \j  =  l,2,3,0,R-ID  . 


(11) 
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Identification  Probability  (Pid)’  Pid  is  conditioned  on  Ptgt-dec,  Poeti  PiD-dec 
and  on  targets  only.  It  measures  the  ability  for  ATR  algorithms  to  positively  identify 
the  object  types. 

E  Mu 


Pid  - 


i=l,2,3,0 


(12) 


E  E 

i=l,2,3,0  \j=l,2,3,0 


Correct  Label  Probability  (Pcl)-  Pcl  is  conditioned  on  Ptgt-dec,  Poet,  PiD-dec, 
targets  as  well  as  non-targets.  It  indicates  all  labeling  results  even  non-target  objects 
labeled  as  target  objects. 


PCL  = 


E  Mu 

i=l,2,3,0 


E 

i=l,2,3,0,4,5 


E 

t  =  l,2,3,0 


Mij 


(13) 


These  are  a  few  summary  performance  measures  that  are  redefined  and  clarihed  by 
the  Air  Force  Research  Laboratory  (AFRL),  the  Comprehensive  Performance  Assess¬ 
ment  of  Sensor  Exploitation  (COMPASE)  Center [1].  With  the  effort  by  COMPASE, 
the  unified  and  clearly  defined  summary  performance  measures  can  effectively  assist 
researchers  in  ATR  community  to  communicate  their  results  and  ATR  algorithm  per¬ 
formance  evaluations.  We  intend  to  use  these  measures  to  evaluate  image  segmentation 
algorithm  performance  in  the  context  of  ATR. 


4  Image  Segmentation  Algorithm  Performance  Mea¬ 
sure  in  the  Context  of  ATR 

In  the  ATR  applications,  the  image  segmentation  algorithms  intend  to  label  the  pixels  as 
“point  of  interest”  or  “point  of  no  interest”  according  to  a  prescribed  feature  space.  Some 
of  the  features  such  as  edges,  target  scattering  centers,  shadows,  etc.,  are  commonly  used 
in  ATR  algorithms.  The  purpose  of  image  segmentation  in  ATR  is  to  label  the  POIs 
and  group  them  into  a  ROI  for  ATR  algorithms  to  label  the  type.  The  ultimate  goal 
of  a  segmentation  algorithm  is  to  provide  POIs  precisely  according  to  the  prescribed 
feature  spaces  such  that  a  positive  identification  can  be  achieved  by  an  ATR  system. 
Some  of  the  ATR  algorithms  may  not  be  very  sensitive  towards  the  preciseness  of  POIs 
and  others  may  well  be.  With  the  variations  of  experimental  bins  and  different  mixtures 
of  operating  conditions,  the  performance  of  different  image  segmentation  algorithms 
will  affect  the  performance  of  various  ATR  algorithms  differently.  To  ensure  a  good 
performance  image  segmentation  algorithm  is  used  in  an  ATR  system,  the  performance 
has  to  be  evaluated  according  to  or  at  least  correlating  to  the  ultimate  performance 
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measure  of  the  ATR  system.  The  performance  of  image  segmentation  algorithms  really 
should  be  judged  in  the  context  of  ATR  applications. 

4.1  Baseline  Algorithms 

To  support  this  notion,  we  hrst  have  to  establish  a  set  of  baseline  ATR  algorithms  that 
uses  a  similar  feature  spaces.  In  this  pursuit,  we  hnd  out  that  even  the  core  of  the  ATR 
community  can  not  agree  on  what  consists  of  baseline  algorithms  due  to  the  complexity  of 
the  ATR  problems.  Some  of  the  ATR  algorithms  are  very  complicated  with  so  many  ad 
hoc  thresholds  and  parameters  to  select  that  it  is  impossible  to  reproduce  the  comparable 
results  as  they  published.  Some  other  ATR  algorithms  do  not  contain  enough  details 
in  the  publication  for  us  to  implement.  However,  we  managed  to  implement  three  ATR 
algorithms  for  conducting  our  experiments.  In  those  three  algorithms,  the  segmentation 
algorithm  required  is  not  more  than  just  locating  a  block  ROI  that  contains  the  POIs 
of  a  potential  target. 

The  hrst  algorithm  is  the  classic  template  matching  method.  It  is  a  very  simple  and 
reliable  ATR  algorithm  that  uses  the  minimum  mean  square  error  (MSE)  between  the 
received  ROI  and  stored  templates  to  classify  the  target  chip  in  class  and  pose[5].  Under 
the  assumption  that  over  a  small  azimuth  angle,  the  radar  signature  of  the  target  remains 
relatively  constant,  the  templates  are  formed  by  averaging  all  the  training  images  in  a 
small  interval  of  azimuth  angle  at  each  azimuth  angle  location.  In  our  experiment,  we 
have  completely  reproduced  the  experimental  results  by  Worrell  and  Parker  [5].  We  then 
did  a  slight  modihcation  on  the  algorithm  by  removing  the  mean  from  all  the  image 
chips.  It  noticeably  improves  the  performance  of  the  algorithms. 


{N-lN-l 

Y  \T{x,y) 

x=Q  y=0 


(Mijix,  y)  -  s(x  -  Xopt,  y  -  yopt))\ 


i=0,-,P-l 


(14) 


where  {c,9)  are  class  and  pose  that  minimize  the  error.  Mij  is  the  template  for  the  ith 
pose  and  jth  class.  There  are  total  of  P  number  of  poses  in  each  class  and  total  of  Q 
number  of  classes.  T  is  a  the  binary  mask  for  the  brightness  of  threshold  that  was  the 
quarter  power  function  in  our  implementation.  s{x  —  Xt,y  —  yt)  is  the  translated  SAR 
image  chip  (ROI)  magnitude. 

The  second  algorithm  uses  the  conditionally  Gaussian  stochastic  signal  modeling 
approach  to  SAR  ATR[6].  In  this  approach,  the  radar  return  is  modeled  as  a  complex 
Gaussian  process 

r  =  s{9,  c)  +  w  (15) 

where  s  is  a  complex  Gaussian  random  vector  conditioned  on  (6*,  c)  with  conditional 
mean  ij,{9,c)  =  0  and  a  diagonal  covariance  matrix  K{9,c).  w  is  a  zero  mean  Gaussian 
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random  noise  with  covariance  matrix  NqI.  The  conditional  covariance  matrix  of  the 
radar  return  r  can  be  found  as 


E{{rr'^\0,c}  =  K{e,c)  +  Wol. 


(16) 


The  Log  likelihood  function  becomes 


l(r\9,c)  =  ^ 


-\n{KiA0,c)  +  No)  - 


Kij{6,  c)  +  No 


(17) 


The  only  unknown  in  evaluating  l{r\6,c)  is  the  values  of  Kij{9,c)  +  No  which  will  be 
estimated  from  the  training  data.  Under  the  assumption  that  the  variance  of  each  pixel 


is  nearly  constant  within  small  intervals,  Wk  = 


2wK 


d  IkK 
2’ 


+  ^ 
^  2 


,  of  the  posing  angle 


9k  =  the  variance  of  ith  pixel  is  estimated  from  SAR  testing  images  of  class  c  as 


cr. 


}\0kNi)  = 


—  y 

e&Wk 


■i{9,  q)  p,  1  <  /  <  t,  l<k  <Ny, 


(18) 


where  Nk  is  the  total  number  of  training  images  within  the  azimuth  interval  Wk  and  t  is 
the  number  of  classes  in  the  database.  The  classification  of  the  ROI  is  performed  using 
Bayesian  approach  by  selecting  the  target  class  c(r)  such  that  the  P(c|r)  is  maximized. 

CBayes(r)  =  ITiaX  ^  ^  P(r  |6'jb,  Sj,  c)  (19) 

"  k  j 


where  Sj  is  all  possible  object  locations. 

The  third  baseline  algorithm  is  developed  by  the  Lincoln  Laboratory [7].  The  algo¬ 
rithm  uses  a  three  stage  processes,  detector,  discriminator,  and  classifier  to  label  the 
type  of  object.  The  detector  is  a  two-parameter  CFAR  detector 

(20) 

where  s{x,  y)  is  the  amplitude  of  pixel  under  the  test,  and  Uc  are  mean  and  standard 
deviation  of  the  clutter  inside  the  boundary  stencil,  and  Kcfar  is  a  threshold  chosen 
to  control  the  false  alarm  rate.  The  boundary  stencil  is  simply  a  area  defined  around 
each  test  pixel.  The  discriminator  process  each  ROI  produced  from  the  detector  and 
rejects  any  ROI  that  does  not  contain  man  made  objects.  The  classifier  stage  consists 
of  a  Mean  Squared  Error  template  matching  classifier.. 

^  N 
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where  N  is  the  number  pixels  in  reference  template,  Ri  are  pixels  in  the  normalized 
reference  template  and  are  pixels  in  the  processed  test  image. 

The  template  matching  algorithm  has  been  implemented  at  AFRL  by  Michael  Bryant 
in  Matlab.  We  just  made  some  miner  changes  (remove  image  mean)  to  generate  best 
ATR  results  with  MSTAR  data.  Based  on  the  template  matching  Matlab  code,  we 
implemented  the  other  two  algorithms  to  form  a  baseline  ATR  algorithm  suite.  All 
these  algorithms  are  only  require  rectangular  shaped  ROI  segments. 


4.2  Preliminary  Evaluation  Suite 

With  the  baseline  algorithms  in  place,  the  evaluation  algorithms  can  be  evaluated  in 
the  context  of  ATR.  The  block  diagram  of  evaluation  suite  is  shown  in  figure  2.  Since 


ATR  sensor 
image  sources 

EO/IR 

SAR 

Hyperspectral 

X-ray 

Ultrasound 

MRI 


Feature  space  in 
the  context  of  ATR 

Temperature 

Edge 

Shadow 

Scattering  center 
Intensity 
invariants 
speed/distance 
Textuer,  etc. 


Image 

segmentation 

algorithms 

under 


Registered  ATR  features  (ROI): 
Shape,  Pattern,  Locations, 
Speed/distance  registration,  etc. 


^  evaluation 

prescribed 

Baseline  ATR 
algorithms 

features 

Figure  2:  The  evaluation  suite  for  image  segmentation  algorithms  in  the  context  of  ATR 

the  scores  from  image  segmentation  evaluation  {pps^  pdh^  and  cip)  and  all  the  scores  for 
ATR  summary  performance  measures  are  normalized  from  0  to  1.  The  score  analysis 
can  be  done  by  various  rank  orders  of  the  score  vector  S. 

S  =  [pps  pdh  cip:Ptgt-dec  Poet  Ps  {^  —  Pfa)  PiD-dec  PiD  Pcl]  (22) 
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Some  of  the  typical  rank  orders  such  as  maximum  value. 

scmax  “  nn3x{S'(z)} , 

% 

(23) 

median 

scmdn  =  median{S'(i)} 

(24) 

and  weighted  mean 

SCav  =  •  S{i)] 

(25) 

I 


where  w{i)  are  selected  weighting  values  with  J2w{i)  =  1.  If  some  of  the  scores  in  score 

i 

vector  S  are  not  needed  in  the  evaluation  report,  then  the  corresponding  weight  can  be 
set  to  zero.  A  evaluation  score  use  the  sum  of  three  top  ranked  scores  in  score  vector  S, 

sCmaxS  ~  T  S(i)  (26) 

max{3} 

We  can  also  use  only  a  subset  of  the  score  vector  for  the  algorithm  evaluation.  For 
example,  the  evaluation  uses  the  partial  sum  of  identification  scores 

SCps-id  =  {PiD-dec  +  PiD  +  Pcl) (27) 


We  did  some  preliminary  evaluation  experiments  using  the  public  released  Moving 
and  Stationary  Target  Acquisition  and  Recognition  (MSTAR)  SAR  image  data.  The 
test  uses  two  targets  (BMP-2  and  T-72)  and  one  confuser  (BTR-70)  that  is  considered 
not  a  target  in  this  experiment. 

The  first  experiment  uses  the  template  matching  baseline  algorithm.  If  the  ROI  is 
segmented  tightly  with  a  size  of  64-by-64,  the  confusion  matrix  is 


BMP-2 

T-72 

Non-target 

BMP-2 

534 

21 

32 

T-72 

17 

481 

84 

BTR70 

108 

9 

79 

(28) 


and  the  corresponding  partial  score  vector  is 

5  =  0.9008  Ps  =  0.9683  (1  -  Pfa)  =  0.4931  Pid  =  0.9639  Pcl  =  0.8675]. 

(29) 

The  confusion  matrix  for  a  segmentation  not  so  tightly  with  a  size  of  80-by-80  is 


BMP-2 

T-72 

Non-target 

BMP-2 

556 

0 

31 

T-72 

0 

497 

85 

BTR70 

122 

2 

72 

(30) 
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and  the  partial  score  vector  is 

5  =  0.9008  Ps  =  0.9624  (1  -  Pfa)  =  0.3673  Pm  =1  Pcl  =  0.8946], 

(31) 

The  score  comparison  list 


^^mdn 

'5Cmax3 

^^ps—id 

64-by-64 

0.9683 

0.9008 

0.83872 

0.9443 

0.9157 

80-by-80 

1 

0.9008 

0.8250 

0.9544 

0.9473 

tells  us  that  the  segmentation  algorithm  segment  the  target  image  chip  not  so  tight  to  the 
boundary  of  target  pixels  (ROI)  is  better  for  the  template  matching  ATR  performance 
in  majority  of  the  scores. 

On  the  other  hand,  the  experiments  on  the  conditionally  Gaussian  baseline  algorithm 
show  different  conclusion.  For  the  segmentation  size  of  64-by-64,  the  confusion  matrix 
is 


(33) 

and  the  partial  score  vector  is 

^  =  0.8516  Ps  =  0.8986  (1  -  Pfa)  =  0.2  Pm  =  0.8710  Pcl  =  0.7136]. 

(34) 

When  the  segmentation  size  of  48-by-48  is  used,  the  confusion  matrix  becomes 


BMP-2 

T-72 

Non-target 

BMP-2 

347 

50 

58 

T-72 

67 

443 

100 

BTR70 

120 

80 

50 

BMP-2 

T-72 

Non-target 

BMP-2 

508 

29 

50 

T-72 

24 

493 

66 

BTR70 

107 

84 

5 

(35) 


and  the  partial  score  vector  is 


p  =  =  0.9009  Ps  =  0.9051  (1-  Pfa)  =  0.0255  Pm  =  0.9497  Pcl 

The  score  comparison  matrix 


0.8040]. 

(36) 


'^Cmax 

^^mdn 

'5Cmax3 

^^ps—id 

64-by-64 

0.8986 

0.8516 

0.7070 

0.8737 

0.7927 

48-by-48 

0.9497 

0.9009 

0.7170 

0.9186 

0.8791 

clearly  shows  the  tight  segmentation  helps  ATR  algorithm  performance. 
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As  to  the  scores  like  pps,  pdh,  and  cip,  they  would  indicate  higher  score,  in  general, 
for  the  segmentations  that  are  tighter  to  the  ROI.  However,  it  is  not  necessary  the 
reflection  of  ATR  performance.  The  experiments  support  the  notion  that  the  evaluation 
suite  to  examine  the  performance  of  SAR  image  segmentation  should  be  conducted  in 
the  context  of  ATR  performance. 

5  Current  Problems  in  Evaluation  and  Road  Map 
to  Future  Study 

In  this  report,  we  have  presented  a  evaluation  suite  to  examine  the  performance  of  SAR 
image  segmentation  algorithms  in  the  context  of  ATR  performance.  The  preliminary 
experiments  show  that  the  evaluation  in  the  context  of  ATR  does  show  some  advan¬ 
tages.  The  composite  scores  from  the  evaluation  suite  do  reflect  the  ATR  performance. 
However,  it  should  be  noted  that  there  are  still  many  problems  associated  with  this 
preliminary  image  segmentation  evaluation  suite. 


5.1  Problems  in  current  evaluation  test 

The  first  problem  is  that  the  cost  associated  with  the  evaluation  suite  is  much  higher 
than  just  the  test  of  image  segmentation  algorithm  without  invoking  the  ATR  algorithm. 
Furthermore,  the  implementation  difficulty  on  various  ATR  algorithms  is  another  high 
cost  item.  Secondly  due  to  the  high  complexity  of  ATR  algorithms,  there  may  be 
many  uncertainties  on  the  implementation  and  on  the  selection  of  parameters  (such 
as  threshold  values)  of  an  ATR  algorithm.  The  uncertainties  will  affect  the  scores  of 
ATR  performance.  It  is  quite  often  that  one  implementation  performs  differently  than 
the  other  implementation.  In  order  to  find  out  the  score  change  due  to  the  difference  in 
image  segmentation  algorithms  not  the  difference  in  the  implementation  versions  of  ATR 
algorithm,  we  need  to  calibrate  the  ATR  algorithms  in  the  evaluation  suite.  For  many 
complicated  ATR  algorithms,  it  is  very  hard  to  achieve.  In  our  experiments,  we  just 
simply  to  adjust  the  parameters  in  the  ATR  algorithms  to  achieve  their  best  performance 
for  a  given  image  segmentation.  It  is  also  costly  and  time  consuming  to  achieve  that. 
The  third  problem  is  again  about  efficiency.  The  task  of  image  segmentation  algorithms 
is  to  discover  all  the  point  of  interest  based  on  prescribed  features  and  group  them 
together  forming  a  region  of  interest.  To  access  the  performance  of  image  segmentation 
algorithms,  we  only  need  to  have  several  ROI  processed.  On  the  other  hand,  the  objective 
of  ATR  algorithms  is  to  label  each  ROI  according  to  the  features  (models)  of  target 
objects.  In  the  experiment  of  testing  ATR  algorithm  performance,  we  often  have  multiple 
ROI  associated  to  one  object  under  different  operating  conditions.  With  multiple  objects 
in  a  test,  we  are  using  much  more  information  than  we  needed  to  judge  the  performance 
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of  an  image  segmentation  algorithm  duo  to  that  the  ATR  performance  evaluation  process 
is  invoked. 


5.2  Road  map  to  future  study 

Current  research  only  concludes  that  the  performance  of  image  segmentation  algorithms 
must  be  judged  in  the  context  of  ATR.  However,  to  make  the  evaluation  process  practical, 
further  research  is  needed  to  improve  the  efficiency  of  the  performance  evaluation  process. 

The  image  segmentation  algorithms  under  the  evaluation  always  have  the  same  fea¬ 
ture  space  as  the  ATR  algorithm  that  receives  the  segmented  ROl  from  them.  What 
we  need  to  find  out  is  a  metric  or  a  set  of  specifications  to  measure  the  quality  of  ROI 
to  an  ATR  algorithm.  The  higher  quality  of  ROI,  the  higher  performance  of  the  ATR 
algorithm.  For  example,  the  ATR  algorithm  using  conditional  Gaussian  model  requires 
a  ROI  with  very  tight  segmentation  boundary  due  to  the  fact  that  ATR  performance  is 
based  on  how  close  the  probability  density  function  (PDF)  of  an  observed  object  to  the 
PDF  of  a  trained  model.  Tighter  segmentation  around  an  object  makes  ROI  containing 
more  pixels  belong  to  one  object.  Therefore,  the  PDF  of  an  observed  object  becomes 
closer  to  the  PDF  of  a  trained  model.  On  the  other  hand,  for  the  template  matching 
method,  the  ROI  of  a  template  model  contains  both  target  object  pixels  and  background 
pixels.  An  overly  tighter  segmentation  can  reduce  the  accuracy  in  the  matching  process. 
The  point  is  different  ATR  algorithms  have  different  requirements  and  different  quality 
metric  for  the  ROTs  segmented  by  image  segmentation  algorithms.  For  an  efficient  im¬ 
age  segmentation  evaluation  system,  we  should  first  generate  a  quality  metric  of  ROI  for 
a  targeted  ATR  algorithm.  Based  on  the  quality  metric,  image  segmentation  algorithms 
can  be  evaluated  and  compared  in  the  context  of  a  specific  ATR  afgorithm.  In  many 
cases,  a  quality  metric  can  be  shared  by  several  ATR  baseline  algorithms  that  have 
the  similar  requirements  on  the  ROI.  Figure  3  illustrates  the  idea  of  an  efficient  image 
segmentation  evaluation  suite  in  the  context  of  ATR. 

The  first  chalienge  that  we  are  facing  is  to  generate  quality  metric  of  ROI  that 
should  be  closely  correlate  to  the  performance  score  of  ATR  algorithms.  This  can  be 
viewed  as  to  establish  an  input  requirements  and  specifications  for  any  ATR  afgorithms. 
We  understand  this  has  not  been  a  standard  process  for  now.  However,  it  could  be  a 
standard  process  for  the  future  development  of  any  object-based  ATR  algorithms.  For 
those  classical  object-based  ATR  algorithms,  we  need  to  develope  and  test  their  quality 
metric  of  ROI.  The  quality  metric  developed  in  the  study  can  be  served  as  standard 
models  for  any  new  projects  in  ATR  algorithm  development.  The  initial  development 
and  test  of  quality  metric  of  ROI  will  be  costly  and  time  consuming  due  to  the  need 
of  invoking  a  complete  ATR  algorithm  evaluation.  However,  with  the  quality  metric  of 
ROI,  any  future  evaluation  of  image  segmentation  algorithms  will  be  very  efficient.  It 
simply  evaluates  the  ROI  generated  by  an  image  segmentation  afgorithm  against  the 
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Figure  3:  An  efficient  image  segmentation  evaluation  suite  in  the  context  of  ATR 

quality  metric  of  ROI  and  generates  scores  and  evaluation  summary.  Therefore,  the 
second  challenge  is  to  develop  a  score  system  using  the  quality  metric  of  ROI.  But  for 
this  part,  we  have  many  prior  work  to  reference. 

As  a  summary  of  conclusion,  we  complete  the  research  by  finding  that  for  ATR 
application,  the  image  segmentation  algorithms  have  to  be  evaluated  in  the  context  of 
ATR.  While,  as  shown  in  our  experiments,  ATR  performance  measures  can  be  used  as 
an  indicator  to  the  performance  of  image  segmentation  algorithm,  the  evaluation  process 
is  very  inefficient  and  high  in  cost.  To  over  come  the  problem  of  inefficiency,  we  planned 
a  future  research  road  map  for  improvement.  The  future  research  in  establishing  a 
quality  metric  of  ROI  for  every  object-based  ATR  algorithms  can  effectively  improve 
the  efficiency  of  image  segmentation  algorithm  evaluation  in  the  context  of  ATR.  The 
concept  is  very  simple  that  the  object-based  ATR  algorithms  have  to  have  a  specific 
measurement  metric  to  quantify  the  quality  of  its  inputs,  ROI.  The  higher  the  quality 
measure,  the  better  the  ATR  performance.  And,  at  the  same  time,  the  output  (ROI) 
from  the  image  segmentation  algorithm  used  in  the  ATR  should  be  measured  using  the 
same  quality  metric  of  ROI  established  by  the  ATR  algorithm. 
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